Software systems such as Enterprise Resource Planning (ERP) systems have become the computational backbone of many business organizations. Such software systems can be quite complex and include a large number of software modules that can be installed, updated, and/or customized specifically for an organization. Validating that the software systems are operating as expected typically requires extensive testing, which can be an expensive and labor-intensive process.
Current testing approaches adopted by many organizations require each organization to devise its own testing suite. Thus, each organization needs to learn which tests are effective, which aspects of the software system should be tested, and how to do so in a cost-effective way. Gaining this knowledge may require much effort and experience; in the meantime, testing the software systems may be a less effective and prolonged process. However, were organizations able to utilize each other's testing-related knowledge, which is in a sense a wisdom of the crowd (of testers), they might be able to come up with a more effective and efficient testing plan. That being said, there are many obstacles in the way of harnessing the wisdom of the crowd when it comes to testing. For one, software systems belonging to different organizations may be each customized for their respective organizations. Thus, it is not likely that a test devised for a first organization will run “as is” on a system belonging to a second organization. Additionally, tests used to test a system belonging to an organization often involve proprietary data related to the organization. Thus, organizations are not likely to share their testing data if it means that in the process their proprietary data is at risk of being leaked to an unauthorized party.
Another problem that hinders utilization of the wisdom of the crowd for testing is the fact that the behavior of some large scale software systems may highly depend on their setup files, which may customize many facets of the software systems. As a result, essentially the same large scale software systems having different setup files may behave so differently that it is difficult, and sometimes impossible, to utilize testing data from one organization for the purposes of another.